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ABSTRACT 



This paper discusses instructional design considerations 
related to learner assessment in multimedia instruction. Topics addressed 
include: current practice in learner assessment; planning and presenting 
instructional objectives; objectives* role in learner orientation; objectives 
and '* Just-in-Time” learning; feedback to the learner during instruction; 
learner assessment via computer- scored testing; online course assessment 
methods; portfolio and project instructor-based learning assessment; and 
durability of final assessment. Contains 17 references. (MES) 
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Become Aware of Current Practice in Learner Assessment 

The designer of instructional multimedia needs to be aware of currently accepted practices to stay 
^praised of the array of methods and ^>plications from which he or she will want to choose and also to be 
aware of practices that ^Tpear to be expected, appropriate, and frmctional. Most experts agree that several 
measures and methods should be used to determine course effectiveness. 

Current computer-based instruction (CBI) and Web-based instruction (WBI) assessment practices 
include a variety of methods such as instructors evaluations of students, students’ self assessments, peer to 
peer assessments, exams, project based assessment, and assessments that are tied into log-in and usage data. 
Bergman and Moore (1990) suggest three questions that designers should answer about the evaluation 
process: Is the evaluation voluntary or required? How much user activity is being assessed? What type of 
feedback is provided? Bergman and Moore believe that voluntary self-assessment "is a sound strategy for 
adults" (p. 63). However, they also list three more traditional categories, common in assessment, that they 
feel are appropriately used for specific circumstances: single items — to test knowledge of single concepts; 
groiq)ed items — for assessing knowledge of "logical units" (i.e., on chunked material); and multiple groups 
of items — for a final test. Nicholas Iiq>pa (1998) makes a useful distinction between test styles, specifically 
concept tests and simulation tests. Concept tests are used to assess learners' understanding, while 
simulations test learners' performance. In discussing how to assess concept and performance via digital 
media, liq>pa cautions that digital assessment has its limits. He says, for instance, that tactile skills are 
"among the very few" types of skills that digital media cannot assess and points out digital media cannot, 
for example, test how smoothly the student manages to screw in a spark plug, which is an important part of 
the process. 

Current evaluation procedures range from “one shot grading, “ such as taking a final or semester 
test, to mastery or ‘^perfection-based” grading, which allows students to redo assignments, within an 
acceptable time fr-ame, until they are perfected (Cox, 1999). Counter to traditional design theory is the 
growing need to specify instructional goals and learning outcomes to accept what is termed as a “diversity 
of outcomes.” Duchastel explains that this can best be achieved by, “aligning evaluation not with 
knowledge, but with task accom plishm ent that utilizes knowledge” (1997, p. 225). 

Planning and Presenting Instructional Objectives 

The most critical issues for instructional designer are those regarding how best to plan, present, 
and evaluate objectives and outcomes for CBI/WBI instructional materials. Bannan and Milheim (1997) 
describe the issue in this maimen 

The World Wide Web is becoming a major source for educational material delivered to learners 

who prefer (or are required) to learn apart from a traditional classroom. While the educational 
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potential of this medium is just beginning to be realized, its utilization will certainly increase over 
time as larger numbers of educators and learners see the significant value in this type of 
instruction. However, while there is tremendous potential for this type of learning, there is also a 
significant need to describe these Web-based courses in terms of their overall instructional design 
characteristics, rather than defining each course only by the specific content it provides. Without 
this organizational process, courses will be perceiv^ and categorized based primarily on their 
subject material, rather than the instructional strategies and tactics used for the delivery of the 
educational material (p. 381).” 

These issues are also concerns of the designer of instructional programs delivered by means of 
CD-ROMs or other computerized stand-alone tutorial materials. In traditional instructional design, 
objectives are the critical, conceptual foundation on which the instructional designer constructs the 
fiameworic of tiie instruction to be. The instructional objectives are usually the starting point, and serve as 
the basis for all other instructional design decisions made during tiie instructional design and development 
process (Dick and Carey, p. 5). As explained by Dick and Carey, tiie objectives est^Iish the purpose of 
instruction in terms of desired learner outcome bdiavior or performance, conditions under which the 
behavior is to be evaluated, and standards for evaluation and assessment. 

In terms of learner assessment, then, objectives teU the learner, at tiie outset, what he or she is 
expected to leam in the course of the lesson. For the instructional designer, identifying the objectives of 
the instruction to be delivered is a critical step in choosing the best cognitive and presentational strategies 
for the instructional material and involves awareness of the learner's needs as well as issues such as 
cognitive styles, need for learner interaction, and other related concerns (p. 126). 

For the mstructional designer who is developing CBI or WBI, instructional objectives £^>pear to be 
as critical as they are to the designer of "traditional” instruction. That is, for the designer, objectives are a 
critical step in (1) the designer's choice of the best cognitive and presentational strategies for the 
instructional materials to be developed as weD as (2) in tiie choice of best media for instructional delivery 
and otiier, similarly critical decisions in instructional development (Pemici and Casati, pp. 246-247). 
Basically, the designer of computer- or Web-based instruction has to know the outcome intended — 
learning desired or task to be achieved — and the assessment method that will be used to evaluate that 
outcome before the designer chooses the best way to inform the learner about those objectives. 

Objectives^ Role in Learner Orientation 

The instructional designer has to plan so that the initial objectives provide, for the learner, clear 
and reasonable expectations about the assessment phase of instruction. In general, because multimedia 
instruction is usually at least one step removed fiom face-to-face teacher-to-student interaction, the 
statement of objectives at tiie beginning of a lesson plays a critical role in learner orientatioa For distance 
learners, the stated initial objectives of an on-line course of study or a CD-deliv^ed tutorial may be the 
chief means of allowing the learner to self-advise in selection of one course of instruction over another 
(Benson, 1994). 

In addition, for "remote" or CBI instruction deliv^ed as "automated" computer-tutorials, initial 
stated objectives have proved to be important as "advance organizers" for learners* outcome performance. 
For example, in an e?qperiment by Beasley and Waugh (1996), g^xsps of college students studying 
computerized units of instruction ^owed remarkably different test scores and significant differences in 
content retention that were affected by only one experimental fector. The experiment^ group of students 
was told that, after viewing the instructional materials, they would be asked to draw a concept m^ of how 
the topics of the lesson related to one another. Knowing tiie objective -r and the means of assessing what 
they were to stu^ — acted as a powerfully effective ’’advance CH’ganizer” in allowing tiie students to view 
the instructional materials in tiie context of tiie performance goal and to perform tiie target task fer better 
than the control groiq>, who were not told anytiiing otiier tiian that tiiey were to "cover" the unit of study. 
Not only did the experimental group — the ones told the objective — perform better than the control group^ 
these differences in assessed pafonnance were still present a week later when the two groiq>s were given a 
"pop quiz” to check their retention of the informatioa Again, the group that was informed of the 
assessment objective performed significantly better th an tiie "unknowing" control groi^. 
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Objectives and ” Just-in-Time” Learning 

The role of learning objectives as indicating perfonnance outcomes and being critical in deciding 
on a course of stuffy is also related to the issue of "just-in-time" learning, which is an important source for 
Web-based learning and instructional design. Just-in-time learning is information delivered "so diat each 
individual may learn just what he or she needs at the time when he or she needs it" (Romiszowski, 1997). 

The "just in time" learning concept is related to the idea of both goal-oriented learning and to 
performance-based assessment It is therefore similarly related to the requirements that objectives and 
assessment outcomes be clearly estabhshed from the beginning of the instructional development process so 
that the instructional designer can choose which cognitive strategies and which presentation and assessment 
methods are most probably going to result in the desired learning. 

Feedback to the Learner During Instruction 

In addition to a final assessment, in the sense of an overall "grade" for the learner who completes 
the unit of multimedia instruction, Bergman and Moore discuss three types of more sophisticated fee^ack 
categories for learner interaction that they beheve should be considered by the instructional designer 
Informative feedback,, friat simply identifies a correct or incorrect response; Reinforcing feedback, that 
praises or critiques (both of vMch should be limited); Diagnostic feedback, that involves comments or 
suggestions about the learner's response. 

Dick and Carey (1996) - agree, beheving that learners need a variety of types of feedback during 
the course of the lesson in order to improve their performance on the final examination for the instructional 
unit (p. 192), which ^pHes whether the instruction is dehvered in the electronic or the traditional 
classroom. Even more important, then, is feedback to the learner using multimedia instructional materials 
in heu of an instructor's providing this type of interaction. 

Learner Assessment via Computer-Scored Testing 

Online or computer-based multimedia instruction can utilize instructor-based evaluation in a 
multitude of forms, just as the face-to-&ce classroom setting affords multiple methods of evaluation. One 
method of evaluation that is unique to CBI and WBI is the option of having the computer score the 
student's woik and record a grade "automatically." 

As more and more emphasis has been placed on behavioral objectives and “statements of exphcit 
behaviors that learners must demonstrate, it has been increasingly obvious that a fair and equitable 
evaluation system is one that measures those specific behaviors. Although it is possible to create computer- 
based testing and scoring, there are many concerns related to this type of evaluation. Grades are grades, 
and students are sometimes ingeniously detennined to improve their grade point averages or acquire credit 
where credit is definitely not due. One solution to the academic honesty issues appears to be admiiiistermg 
“final" exams in proctored cQnq>uter labs. Other possible ^>proaches to the issues of academic honesty aixl 
reliability raised in relation to assessment of student learning in CBIAVBI instructional modules may 
involve use of frte technology that has introduced the problem in the first place. For example, some virtual 
colleges and online universities are discussing use of security and identity-verification methods, such as 
voice recognition or fingerprint-recognition systems. 

Online Course Assessment Methods 

Two basic types of assessment for online college-level courses prevail: (1) those with CBI and 
"automated" or computer-adininistered types of instruction, the most typical final assessment method is the 
proctored exam, and (2) those with instructor-monitoied distance learning types of instruction, with the 
most typical assessment method being criteria-based evaluation, with a final grade assigned by the 
instructor (Rasmussen er u/, 1997). 

Instructor-monitored distance learning courses tend to follow one of two models: (1) the 
"portfoHo" or "contract" model, with students turning in Sequent, small reports or conq>leting a scheduled, 
sequence or series of assignments, and (2) the project model, in which the student turns in, at the end of the 
course, one big project or term paper (Hudspeth, 1997). 

An example of typical statements about proctored tests and exams is this one fiom Indiana 
University's online Chemistry 101 course: 



Quizzes and Exams preferably can be taken on tbe Internet This will require that you have a 
proctor with a computer system that meets the system requirements to admini ster the exam or 
quiz. Your proctor should be associated with a reliable office or agency in your community 
(employment siq)ervisor, library, high school, bank, office, business personnel, etc., (no relatives 
or personal friends). If this is not possible, a second option is to find a proctor with faxing 
capabilities.... We suggest you make arrangements with your proctor and have him/her complete 
and return a Proctor Agreement as soon as possible (by or before the orientation meeting). This 
signed agreement is a necessary prerequisite. 

( http://weblab. iupui .edu/oncourse/courses/c 101 demo/ ) 

Clearly, age-old concerns about ethical practices in assessment (i.e., cheating) take on new twists 
in the distance-learning environment. The issue of authorship of student work has always been one that is 
difficult to resolve, even when the course is taught with traditional methods. A statement about 
expectations for online students, again frx>m Indiana University, illustrates a common philosophical 
q>proach to the problem of authorship: 

Much of American life operates on the honor system — and so do a number of areas of this 
course. Thus the laboratory work and write-^, the homework assignments are done away from 
University supervision and are expected to be performed by you and you alone. Violation of this 
e?q)ectatiQn ultimately hurts you more than anyone else. 

( http://weblab.iupui.edu/oncourse/courses/cl01demo/ ) 

Portfolio and Project Instructor-Based Learning Assessment 

During the Web Based Tr aining Online Learning ’99 conference. Brad Cox, author of the award 
winning. Taming the Electronic Frontier, was asked how he evaluated instruction in his online course. He 
replied: 



Briefly, I rely primarily on experiential learning (action learning). Each task presents instruction, 
invites the student to put it in practice and report the results in the cont^t of the material that 
preceded this task. Some tasks have students read web-based or paper-based materials, summarize 
what it says, and demonstrate that they have ^lied each lesson to the web-based portfolio. Other 
tasks, such as the desert crash simulation, portfolio peer assessment, and web-based sociometic 
tasks, take a more quantitative zqjproach. Each student produces a web-based portfolio and 
participates in a semester project These provide considerable insight into how well the student is 
doing, both by me and the student’s peers. 

Cox's ^roach recommends using a series of small, sequential, individualized tasks and student- 
centered person^ re^xmses to provide multiple checkpoints during the online course ^d ensure that 
students, in order to complete the assignments, have to keep up with the class readings and respond to class 
assignments themselves. Multiple, individualized tasks are harder to counterfeit because of &e necessary 
coordination and planning involved for the student to arrange for someone else to do the woik in a timely 
and appropriately specific mann er 

Cox suggests a sequential, portfolio-style ^q^proach to the student's progress through a course and 
assessment of that progress. Another, somewhat similar, ^jproach is recommended by DeLayne Hudspeth 
(1997) in discussion of "just in time" learning programs. Hudspeth (p. 356) recommends a "contract" 
approach, which he describes as: 

A written agreement, a learning contract, which defines what the instructor is proposing and what 
the student agrees to do. The initial syllabus might describe 20 possible assignments of which the 
learner must provide the first six or eight (the "must know" content), but could select another six 
or eight "nice to know" outcomes from an extended list. The final list could reflect individual 
interests or different career backgrounds. Peiiiaps two or three unique outcomes could be 
negotiated with the instructor. If a grade system is used, then a predetermined number of points 
could be required for a specific grade. 
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Hudspeth points out that the benefit of using a contract-type approach is that the course 
assignments and the criteria for their completion can be e;q>licitly describe for the student, but the student 
can use considerable flexibility in e;q>loring issues and resoiirces that have personal interest This approach 
should encourage the student to see the benefit of actually doing the learning rather than trying to find a 
stand-in (p. 356). 

Another method of online course assessment, particularly for adult learners, is the con^letion of 
one large end-of-course project (Mood, 1995). For example, in a business course, students may submit, at 
the end of instruction, a fhlly developed business plan or, as the final project in an online course, the 
student may be required to submit a substantially researched term paper. 

With project-based assessment, the dangers of having Ae student simply purchasing a pre- 
packaged term paper are diminished the more individually the project is tailored to the resources used in the 
course, the studenf s individual interests, and the use of intermittent "checkpoints" — such as weekly, brief, 
e-mailed "progress reports" that are individual and specific enough to reduce the likelihood of students' 
thinkin g they can submit someone else's work and avoid being found out (Hudspeth, p. 354). 

Others suggest using Internet-based videoconferencing or online synchronous class discussions or 
similar performance-on-demand type class activities as part of authentic or performance assessment of 
student achievement. In addition to encouraging participatory learning, enhancing student communication 
and/or technology skills, and interaction between isolated learners or between learners and the distant 
instructor, these "real-time" performance activities also tend to ensure, incidentally, that the actual student 
be present and accountable for course content. 

Durability of Final Assessments 

^Tinal” evaluation for certification or course credit is unlikely to be e liminat ed anytime in the near 
future. However, the advent of the technology that has provided the means to offer computer-based or 
online courses of study has also caused the le-evaluafion of assessment issues, such as what should be 
assessed in a virtual, fbr-credit course. Many issues are currently being debated, including how to best 
create learning enviromnents and evaluation procedures that encourage the s^lication of knowledge rather 
than rote memorizatiQn; how to design instruction to help learners to think for themselves; and how does 
computer based learning works in conjunction with other resources in the planning, in^lementing, 
evaluating and improving loop? (Harvey, 1998) 

In discussing learner assessment, the role of the instructional designer is growing. Because the 
instructional designer is intimately involved in establishing the instructional objectives for instruction as 
well as the assessment instruments by \^ch the student's learning will be judged, the instructional 
designer's role is critical to insuring th^ CBl/WBI instruction works as expected. Since the instructional 
delivery is often remote, and results are sometimes difficult to evaluate in terms of instructional 
effectiveness, the unique position of the instructional designer is more critical to the development and 
evaluation of technology-based instruction than ever. 
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